Russ Thurow USDA Forest Service Rocky Mountain Research Station
Claire McGrath Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, riverbio@yahoo.com
Kevin See Biometrician, Biomark Inc, Boise, ID, Kevin.See@merck.com
Salmon redd counts are widespread method to estimate the number of returning adult spawners. However, despite its prevalence in the Northwest, the reliability of redd counts is unknown. This work is focused on developing a statistical model to estimate the observer error in redd surveys, using a variety of covariates related to the habitat and the observer. We described three types of observer error:
Possible covariates in each error model are shown in Table 2.1. To make comparisons with AICc, the random effects must be identical across all models. Therefore, we ensured that the random effect of year was added to any model that didn’t have it.
| Type | Covariate | Air | Ground |
|---|---|---|---|
| Random | Reach | X | X |
| Random | Surveyor | X | |
| Random | Year | X | X |
| Fixed | Alluvium | X | X |
| Fixed | AveBadCond | X | |
| Fixed | AveCanopy | X | X |
| Fixed | AveDepth | X | X |
| Fixed | AveSunny | X | |
| Fixed | AveWidth | X | X |
| Fixed | EscapeEst | X | X |
| Fixed | Experience3 | X | |
| Fixed | I(AveDepth^2) | X | X |
| Fixed | Lithology | X | X |
| Fixed | LYabund | X | X |
| Fixed | LYabund:PeakQ | X | X |
| Fixed | PeakQ | X | X |
| Fixed | redd_dens_obs | X | X |
| Fixed | Slope | X | X |
All covariates were z-scored, and all models were fit using the glmer or lmer functions from the lme4 package (Bates et al. 2015) in R software (R Core Team 2019). The amount of variation explained by fixed and random effects was calculated using the methods of Nakagawa and Schielzeth (2013). Using estimated predictions of the rates for omission (\(\hat{\omega}\)), commission (\(\hat{\eta}\)) and net error (\(\hat{\gamma}\)), we predicted the number of actual redds by either dividing the observed counts, \(c\), by estimates of net error, or by multiplying the observed counts by 1 - estimated rate of commission, and then dividing by 1 - estimated rate of omission.
We performed a cross validation by dividing each survey type data into 10 training datasets where 10% of the data was withheld for testing, and then fitting the naive and best AICc model formulations to the remaining data, and then using those fits to predict the error rates and true number of redds for each survey in the year that had been withheld.
\[ \begin{aligned} redds_{ne} &= \frac{c}{\hat{\gamma}} \\ redds_{om} &= c * \frac{1 - \hat{\eta}}{1 - \hat{\omega}} \end{aligned} \]
The observed error rates are showin in Figure 2.1.
Figure 2.1: Observed error rates.
The model coefficients of the full, best (by AICc) and model averaged models are shown in Table 3.1 and 3.2.
| Survey | Resp | Covariate | avg | best | full |
|---|---|---|---|---|---|
| Ground | Com | (Intercept) | -1.603 | -1.611 | -1.611 |
| Ground | Com | AlluviumN | 0.588 | 0.588 | 0.588 |
| Ground | Com | AveCanopy | 0.040 | 0.040 | 0.040 |
| Ground | Com | AveDepth | 0.170 | 0.170 | 0.170 |
| Ground | Com | AveWidth | -0.143 | -0.143 | -0.143 |
| Ground | Com | EscapeEst | -0.314 | -0.314 | -0.314 |
| Ground | Com | Experience3.L | -0.242 | -0.242 | -0.242 |
| Ground | Com | Experience3.Q | 0.275 | 0.275 | 0.275 |
| Ground | Com | I(AveDepth^2) | 0.051 | 0.051 | 0.051 |
| Ground | Com | Lithology2 | 0.389 | 0.389 | 0.389 |
| Ground | Com | Lithology3 | 0.328 | 0.328 | 0.328 |
| Ground | Com | Lithology5 | 0.324 | 0.324 | 0.324 |
| Ground | Com | LYabund | -0.225 | -0.225 | -0.225 |
| Ground | Com | LYabund:PeakQ | -0.741 | -0.741 | -0.741 |
| Ground | Com | PeakQ | -0.683 | -0.683 | -0.683 |
| Ground | Com | redd_dens_obs | 0.444 | 0.445 | 0.445 |
| Ground | Com | Slope | 0.052 | 0.052 | 0.052 |
| Ground | Net | (Intercept) | -0.258 | -0.249 | -0.845 |
| Ground | Net | AlluviumN | -0.184 |
|
0.257 |
| Ground | Net | AveCanopy | -0.041 |
|
0.023 |
| Ground | Net | AveDepth | 0.066 |
|
0.075 |
| Ground | Net | AveWidth | -0.083 |
|
-0.028 |
| Ground | Net | EscapeEst | -0.080 |
|
-0.080 |
| Ground | Net | Experience3.L | 0.318 |
|
0.294 |
| Ground | Net | Experience3.Q | -0.169 |
|
-0.180 |
| Ground | Net | I(AveDepth^2) | 0.008 |
|
-0.007 |
| Ground | Net | Lithology2 | 0.345 |
|
0.551 |
| Ground | Net | Lithology3 | 0.251 |
|
0.455 |
| Ground | Net | Lithology5 | 0.422 |
|
0.556 |
| Ground | Net | LYabund | 0.092 |
|
-0.036 |
| Ground | Net | LYabund:PeakQ | 0.040 |
|
-0.177 |
| Ground | Net | PeakQ | -0.031 |
|
-0.163 |
| Ground | Net | redd_dens_obs | 0.143 | 0.143 | 0.189 |
| Ground | Net | Slope | -0.047 |
|
-0.055 |
| Ground | Omi | (Intercept) | 0.301 | 0.301 | 0.301 |
| Ground | Omi | AlluviumN | -0.049 | -0.049 | -0.049 |
| Ground | Omi | AveCanopy | -0.027 | -0.027 | -0.027 |
| Ground | Omi | AveDepth | 0.107 | 0.107 | 0.107 |
| Ground | Omi | AveWidth | -0.176 | -0.176 | -0.176 |
| Ground | Omi | EscapeEst | -0.029 | -0.029 | -0.029 |
| Ground | Omi | Experience3.L | -0.599 | -0.599 | -0.599 |
| Ground | Omi | Experience3.Q | 0.608 | 0.608 | 0.608 |
| Ground | Omi | I(AveDepth^2) | -0.023 | -0.023 | -0.023 |
| Ground | Omi | Lithology2 | -0.745 | -0.745 | -0.745 |
| Ground | Omi | Lithology3 | -0.704 | -0.704 | -0.704 |
| Ground | Omi | Lithology5 | -0.741 | -0.741 | -0.741 |
| Ground | Omi | LYabund | -0.186 | -0.186 | -0.186 |
| Ground | Omi | LYabund:PeakQ | -0.339 | -0.339 | -0.339 |
| Ground | Omi | PeakQ | -0.053 | -0.053 | -0.053 |
| Ground | Omi | redd_dens_obs | -0.126 | -0.126 | -0.126 |
| Ground | Omi | Slope | 0.336 | 0.336 | 0.336 |
| Survey | Resp | Covariate | avg | best | full |
|---|---|---|---|---|---|
| Air | Com | (Intercept) | -1.355 | -1.327 | -4.067 |
| Air | Com | AlluviumN | 0.562 |
|
1.491 |
| Air | Com | AveBadCond | 0.003 |
|
0.011 |
| Air | Com | AveCanopy | -0.151 |
|
0.291 |
| Air | Com | AveDepth | 0.006 |
|
0.175 |
| Air | Com | AveSunny | 0.345 | 0.345 |
|
| Air | Com | AveWidth | 0.289 |
|
0.347 |
| Air | Com | EscapeEst | -0.291 |
|
-0.291 |
| Air | Com | I(AveDepth^2) | -0.035 |
|
-0.053 |
| Air | Com | Lithology2 | 1.533 |
|
2.804 |
| Air | Com | Lithology3 | 0.550 |
|
0.559 |
| Air | Com | Lithology5 | 1.305 |
|
2.863 |
| Air | Com | LYabund | 0.442 |
|
0.431 |
| Air | Com | LYabund:PeakQ | 0.314 |
|
0.188 |
| Air | Com | PeakQ | -0.324 |
|
-0.488 |
| Air | Com | redd_dens_obs | 0.069 |
|
-0.034 |
| Air | Com | Slope | -0.179 |
|
-0.370 |
| Air | Net | (Intercept) | -0.195 | -0.183 | -1.196 |
| Air | Net | AlluviumN | -0.222 |
|
0.405 |
| Air | Net | AveBadCond | 0.003 |
|
0.027 |
| Air | Net | AveCanopy | -0.148 |
|
0.003 |
| Air | Net | AveDepth | 0.078 |
|
0.118 |
| Air | Net | AveSunny | 0.220 | 0.220 |
|
| Air | Net | AveWidth | 0.128 |
|
0.112 |
| Air | Net | EscapeEst | -0.089 |
|
-0.089 |
| Air | Net | I(AveDepth^2) | -0.016 |
|
-0.023 |
| Air | Net | Lithology2 | 0.563 |
|
0.938 |
| Air | Net | Lithology3 | 0.199 |
|
0.318 |
| Air | Net | Lithology5 | 0.529 |
|
1.069 |
| Air | Net | LYabund | 0.134 |
|
0.108 |
| Air | Net | LYabund:PeakQ | 0.158 |
|
0.086 |
| Air | Net | PeakQ | -0.084 |
|
-0.158 |
| Air | Net | redd_dens_obs | 0.109 |
|
0.058 |
| Air | Net | Slope | -0.099 |
|
-0.095 |
| Air | Omi | (Intercept) | -0.528 | -0.528 | 0.050 |
| Air | Omi | AlluviumN | -0.082 |
|
-0.082 |
| Air | Omi | AveBadCond | -0.007 |
|
-0.007 |
| Air | Omi | AveCanopy | 0.079 |
|
0.079 |
| Air | Omi | AveDepth | 0.006 |
|
0.006 |
| Air | Omi | AveSunny | -0.257 |
|
|
| Air | Omi | AveWidth | -0.131 |
|
-0.131 |
| Air | Omi | EscapeEst | 0.151 |
|
0.151 |
| Air | Omi | I(AveDepth^2) | -0.004 |
|
-0.004 |
| Air | Omi | Lithology2 | -0.921 |
|
-0.921 |
| Air | Omi | Lithology3 | -0.934 |
|
-0.934 |
| Air | Omi | Lithology5 | -0.280 |
|
-0.280 |
| Air | Omi | LYabund | 0.185 |
|
0.185 |
| Air | Omi | LYabund:PeakQ | 0.132 |
|
0.132 |
| Air | Omi | PeakQ | 0.260 |
|
0.260 |
| Air | Omi | redd_dens_obs | -0.546 | -0.546 | -0.560 |
| Air | Omi | Slope | 0.159 |
|
0.159 |
The relative importance of each covariate in each model is shown in Figure 3.1, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3.2. Observed versus predicted rate plots are shown in Figures 3.3, 3.5 and 3.7.
Figure 3.1: Relative importance of each covariate in ground-based observer error models
Figure 3.2: How much variance in the model response is explained by the fixed and random effects in the best AICc model.
Figure 3.3: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.4: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).
Figure 3.5: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.6: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).
Figure 3.7: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.8: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).
We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects) (Figure 3.9).
Figure 3.9: Bias in predicted ground error rates from cross validation results.
For ground-based surveys, both methods provided fairly unbiased estimates of the true number of redds (Figure 3.10), although the omission/commision models had slightly higher absolute and relative bias (Table 3.3).
Figure 3.10: Boxplots of absolute and relative bias for each type of predictive model.
| Model | Median # Obs. Redds | Median # True Redds | Median Adjustment | Median Abs. Bias | Median Rel. Bias (%) | RMSE |
|---|---|---|---|---|---|---|
| Best Net Error | 36 | 38 | 3.9 | 0.1 | 0.3 | 18.8 |
| Best Omis / Comm Error | 36 | 38 | 4.8 | 0.6 | 1.0 | 16.2 |
| Naive Net Error | 36 | 38 | 3.3 | -1.0 | -2.1 | 19.5 |
| Naive Omis / Comm Error | 36 | 38 | 4.1 | -0.2 | -0.6 | 18.2 |
| Observed | 36 | 38 |
|
-2.0 | -8.0 | 18.9 |
Figure 3.11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.
The relative importance of each covariate in each model is shown in Figures 3.12, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3.13. Observed versus predicted rate plots are shown in Figures 3.14, 3.16 and 3.18.
Figure 3.12: Relative importance of each covariate in ground-based observer error models
Figure 3.13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.
Figure 3.14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).
Figure 3.16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).
Figure 3.18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).
Figure 3.19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).
We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects) (Figure 3.20).
Figure 3.20: Bias in predicted air error rates from cross validation results.
For air-based surveys, both methods provided estimates of the true number of redds that were biased high (Figure 3.21). However, the net error models had lower absolute and relative bias, as well as a smaller root squared mean error (RMSE) (Table ??).
Figure 3.21: Boxplots of absolute and relative bias for each type of predictive model.
| Model | Median # Obs. Redds | Median # True Redds | Median Adjustment | Median Abs. Bias | Median Rel. Bias (%) | RMSE |
|---|---|---|---|---|---|---|
| Best Net Error | 32 | 38 | 6.4 | 1.7 | 9.5 | 24.5 |
| Best Omis / Comm Error | 32 | 38 | 10.3 | 0.8 | 1.3 | 24.2 |
| Naive Net Error | 32 | 38 | 5.8 | -0.5 | -3.2 | 26.3 |
| Naive Omis / Comm Error | 32 | 38 | 7.8 | -1.2 | -3.1 | 27.7 |
| Observed | 32 | 38 |
|
-6.0 | -18.2 | 28.7 |
Figure 3.22: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.
For both ground- and air-based surveys, across all three types of models, the best AICc model predictions and the model averaged predictions were very similar, so although I didn’t include model averaged predictions in my comparisons, I expect them to be very similar to the best model. I chose to use the best AICc model because it would be simplier.
Models for both types of surveys tended to provide unbiased predictions of the number of redds in the stream. Ground-based surveys required slightly less of an adjustment (in either direction) than air-based surveys, but there was certainly an adjustment in both cases, highlighting the need for such observer error models. The median absolute bias for both types of surveys and for both modeling approaches was less than one redd, compared to an undercount of 2 redds for ground-based surveys, and 6 for air-based surveys.
For the ground-based surveys, the most important covariates to explain net error were observed redd density and observer experience, whereas most of the possible covariates had similar importance for omission and commission models. However, the predictions of total redds are almost identical regardless of whether one uses the net error or omission / commission models, or the best model by AICc, or the naive model (only random effects). This suggests that the fixed effects covariates included in this study are not explaining much of the variation in error rates, but that there is information in the random effects of reach and surveyor sufficient to make unbiased predictions. However, those random effects cannot be easily carried on to another study area, or even another year, which limits their usefulness as a predictive model.
For the air-based surveys, AveSunny figured prominently in net error and commission models, whereas observed redd density was clearly the most important for explaining omission errors. The net error and omission / commission models, either the best AICc or the naive versions, all made very similar unbiased predictions, although they appear to slightly under-predict at higher number of redds.
The fact that the predictions from the naive model (random effects only) are highly correlated with the best and model averaged versions for all the various error models, except for net errors for air surveys, suggests that the fixed effects are not explaining much of the observed variation in error rates. This is also seen in Figure 3.2 and 3.13. The root mean squared error (RMSE) of predictions is slightly smaller when using the best AICc model compared to the naive model, so there is some benefit to those fixed effects.
Bates, D., Mächler, M., Bolker, B., and Walker, S. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1): 1–48.
Nakagawa, S., and Schielzeth, H. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2): 133–142. Wiley Online Library.
R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
Biometrician, Biomark, Inc., Kevin.See@merck.com↩︎
Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, riverbio@yahoo.com↩︎
USDA Forest Service Rocky Mountain Research Station↩︎